2 links tagged with all of: decision models + calibration
Click any tag below to further narrow down your results
Links
The author compares newer decision models against LLM judges for automated evals on a leasing-agent failure task. The main takeaway is that the default 0.50 threshold often leaves performance on the table, and that the scores need tuning and calibration before you trust them.
- The default 0.50 decision threshold can miss nearly all failures; tuning it per model, using F1 with an 80% precision/recall floor, improves results.
- OpenAI Decisions had the best separation (F1 about 0.87) but cost about 4.5 times more than Perplexity Decider (F1 about 0.79) per 100 eval runs.
- Raw model probabilities are often poorly calibrated; Platt scaling helped Perplexity, Liquid d1, and Clef Flash, but made OpenAI's slightly worse.
Kev is a family of small language models (0.8B to 9B parameters) built on Qwen3.5 that make yes/no, multiple-choice, and rating decisions on text inputs. You can run them locally on your own hardware with an API compatible with TypeSafe's System One, plus there's a web playground for testing.
- Three model sizes available with published accuracy metrics: Kev-9B reaches 85.2% accuracy on unseen datasets and trails the proprietary Jev model by only 3.5 points, while Kev-4B and Kev-0.8B offer smaller footprints for different use cases.
- All models run on consumer hardware (CUDA, ROCm, Apple Silicon) with the 4B and 9B versions fitting on a 32GB Mac in bf16 precision, plus you can train your own weights using provided code and evaluation data.
- Probabilities are calibrated by default and the models output confidence scores alongside answers, reducing confident errors to 4% on new data compared to 8.7% without calibration.